Scalable Data Space Partitioning In High Dimension

A fundamental process in data mining is to approximate the joint distribution of a multivari-ate data set. A typical approach is to use histograms. However, conventional histograms (with dimension-normal data partitions) create an exponential number of partitions as the number of dimensions increases. In previous work, we presented the DataSphere method for data space partitioning and used it for several types of analysis. A DataSphere creates O(d) partitions on d-dimensional data. In this paper, we generalize DataSphere partitioning to create any polynomial number O(d k) of partitions where k ranges from 1 to d by using hyperpyramids and hyperspheres. Each of these partitioning schemes is hierarchical, allowing data cube-like roll-up and drill-down data analysis. We present several examples of how these partitioning schemes can be used, including visualization.